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ABSTRACT 


Methods of improving image interpretation system output through 
use of interpreter proficiency as a criterion for making interpreter 
personnel assignments were investigated. An experiment was conducted 
to determine if either of two personnel assignment methods using inter- 
preter proficiency as the assignment criterion would yield significantly 
improved team performance. No significant difference in performance due 

| 

to either of the methods tested Were fouhd. A second experiment was 
conducted to determine if assigning the more difficult imagery to the 
more proficient interpreter would result in higher team performance than 
random assignment of imagery to team members. Analysis indicated no 
Significant differences in interpreter performance due to either of the 
methods tested. 


The image interpreter personnel assignment problem was formulated 


as a linear integer program. 
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CHAPTER I 
INTRODUCTION 


Image interpretation is one of the best sources of tactical and 
strategic intelligence. Rapid availability of such intelligence is 
becoming increasingly important in order to counter the mobility of enemy 
forces and to utilize fully the rapid strike capabilities of our forces. 
New sensors, platforms, transmission systems, and "real time” systems are 
being developed which generate large volumes of imagery from which human 


interpreters must extract accurate, timely, complete, and relevant infor- 


mation. This has prompted research efforts directed toward improving 


speed and quality of image interpretation, 


I. STATEMENT OF THE PROBLEM 


In order to meet future image interpretation requirements it is 
necessary to find ways to improve and increase image interpretation out- 
put. This can be accomplished by training more and better interpreters, 
improving the performance of interpreters, or making image interpretation 
tasks less demanding on human interpreters. The purpose of this study was 
to evaluate two possible methods of improving interpretation output. 

Studies sponsored by the U. S. Army Behavioral Science Research 
remeedeerss a) formerly known as the Army Personnel Research Office, 


indicated that having interpreted imagery checked by another interpreter 


resulted in improvement in certain measures of interpreter output. It 


seemed reasonable to assume that interpreters at an interpretation 





facility would differ in proficiency and that often the relative profi- 
ciency of available interpreters would be known. If a check procedure 
was decided upon and available interpreters could be ranked according to 
proficiency, task assignments could be made either with or without regard 
to interpreter proficiency. Given a two-man team, it appeared reasonable 
that assignment of the initial interpretation task to the lower profi- 
ciency interpreter and having the higher proficiency man do the checking 
would yield higher output than other: possible procedures. Experiment I 
was designed to determine if this was true. 

Research has been directed toward development of pre-processing 
techniques that would provide the interpreter with "advance" information 
on interpretability of imagery. Such information would be worthwhile if 
its use resulted in improved interpreter performance. If imagery 
were pre-processed in such a way that its difficulty of interpretation 
was known in advance of human interpretation, assigning higher proficiency - 
interpreters to the more difficult imagery might improve output. Experi- 


ment II was designed to determine if this was true. 
IL. IMPORTANCE.OF THE STUDY 


Several studies of human factors in image interpretation have been 
initiated since about 1961. Many of the experimental results are not 
in consensus with the ithepe interpretation community; few of these experi- 
mental results can be considered definitive. Some studies have recommended 


procedures whose feasibility 1s questionable on account of military consi- 


derations or time costs. This study investigated two no-cost, easily 


re 





implementable interpretation procedures which, if found justified by 


controlled experimentation, should pose no major feasibility problems. 





CHAPTER IT 
REVIEW OF THE LITERATURE 


Interpreter performance is dependent upon characteristics of the 
imagery, pre-processing of the imagery, training of the interpreters, 
previous experience and attained level of competence of the interpreters, 
tactical and strategic information available, equipment available, inter- 
pretation procedures, and personnel organization. Research has been’ 


undertaken in most of these areas. 
I. CHARACTERISTICS OF IMAGERY 


The overall quality of photographic and other imagery is improving, 
due to technological advances; yet imagery quality varies because 
of variations in conditions under which the imagery is made. A study 
by Applied Pigehonags Corporation (2) found that the increase in complete- ae 
ness over time became greater as the quality of imagery was improved. 
Poor quality imagery yielded negligible increases in completeness oven 
time. It was suggested that imagery below certain quality levels need 
not be interpreted. 

Aerial photographs can be made such that targets present either 
vertical or oblique aspects. Studies have been made to determine the 
effects of vertical only, oblique only, and both vertical and oblique. 
In a pilot study, J. E. Ranes(16) found that simultaneous use of 


both views--vertical and oblique--of the target area yielded no sig- J 


nificant improvement over the vertical view alone. The interpretation 





oie 


task was limited to identification of vehicles in convoy. Results of 
another stuay'?) indicate that for mensuration and plotting, vertical 
views should be used. For objects with major dimensions in the vertical 
plane, oblique views should be used. Test imagery was limited to one 
vertical and one oblique view each of a bridge and an airfield, for a 
total of four photographs. Defending the use of both aspects, R. N. 
Colwer1 (19) | an eminent member of the photo interpretation community, 
cited examples where both vertical and oblique views were necessary for 
correct interpretation of objects. 

In the same article Professor Colwell also defended the use of 
stereo imagery, that is,two photographs of the same area taken by two 
cameras a small distance apart. The resulting dual photographs are 
presented to the interpreter in such a way that he is able to use the 
stereoscopic parallax to obtain three-dimensional information. Schwartz 
and Jetaner 19) , however, found no significant difference between stereo 
and non-stereo viewing, Their measures of effectiveness were number right 
and number wrong. No consistent pattern or trend was found to indicate 
superiority of either stereo or non-stereo viewing. They suggested select- 


ive use of stereo viewing. 
II. PRE-PROCESSING OF IMAGERY 


Interpreter performance might be improved if it were possible to 
pre-process the imagery in such a way as to reduce the human interpreta- 


tion task. Ultimately, this would mean complete non-human image 


interpretation. Research efforts have been made in that direction. 





Readers interested in efforts to automate photo interpretation 
are referred to W. S. Holmes' paper, "Automatic Photo Interpretation . 
and Target Location," in IEEE Proceedings, Vol. 54, No. 12, Dec. 1966, . 
pp. 1679-86, which cites ter references. The attainment of 
complete automation is not envisioned in the immediate future; however, 
limited automatic assistance appears to be within the capability of 
current technology. . 

One type of automatic assistance which has been investigated is 
automatic quantification of image quality. If, as several researchers 
have assumed, interpretability is dependent upon image quality, then 
knowledge of image quality might be used to predict image interpretability. 
Cornell Aeronautical ibekretory\ 4) has developed reliable microdensito- 
metric techniques for measuring and specifying contrast, resolution, edge 
sharpness, and granularity directly from photographic imagery. Presumably 
these could be used to quantify image quality and eliminate poor quality ss 
imagery as uninterpretable. _ Measurement of band-widths associated with 
transition from one tone to another in photographic imagery has been demon- 
strated by Minneapelie-Bneyweia 2" ? to be a convenient, reliable, and 
objective method for estimating the ground resolution of photography. 

Manual determination of image quality might be helpful, especially 
if it did not require highly trained personnel. A catalog fectmi gas ®? has 
been developed by which interpreters compare their imagery with catalog 
imagery and assign predicted interpretability values. Discrimination of 
target areas from non-target areas by the catalog technique was correlated « 
with results of actual interpretations, yielding correlations of .77 for 


trained interpreters and .70 for untrained personnel. Correlations of 


predicted accuracy of target identification with interpreted accuracy of 








target indentification were .54 and .51,respectively. Average time per 
test image was 45 seconds. The close correlation values between trained 
and untrained interpreter performance suggests this might be an effective 
way to reduce the work load of photo interpreters by using less skilled 
personnel. 


(14 ) 


Another approach to predicting interpretability is to record 
various data from an initial interpretation and from these compute probable 
accuracy of the interpretation and probable utility of future search. This 
information could form the basis of a decision rule to indicate whether 
or not the imagery should be check interpreted. 

Pre-processing might also take the form of automatic enhancement 


(7) 


of image interpretability. One technique involves obtaining a video 
signal from a transparency and adding to this signal its negative second 
derivative. This so-called "differentation enhancement technique” 
appeared to improve performance principally by increasing the number of 
correct responses, and, to a lesser extent, by decreasing the number of 
incorrect responses. It has been found to be better suited for more 
difficult imagery. 


(5) 


A Boeing study recommended that interpreters view alternately 
flashing superimposed photographs of the same area taken at two different 
times. This technique causes an apparent motion of elements in the 


photography which changed during the time interval between exposures. 


As noted in the study, the effectiveness of the technique is dependent 


upon the amount and complexity of background image disparities. 





III. FEEDBACK INFORMATION AVAILABLE TO INTERPRETERS 


Feedback information can come either from external intelligence 
sources or from the imeibe tnterpretation operation itself. A. E. 

Castelnovo (16) investigated the effects of different levels of externally 4 
provided intelligence information on photo interpretation and found that, 

for a Sequence of imagery, increased information aided the photo inter- 

preters initially, but after a short period of time it had no effect. 

He pointed out that the negative effects of erroneous intelligence must: 

also be considered--something he did not measure experimentally. This 

suggested the possibility that an increased amount of accurate intel- 

ligence information might not necessarily be of significant value in the 

long run. 

Photo interpreters commonly assign subjective confidence estimates . 
to their interpretations. The reliability of these estimates varies. 
Measured reliability for completeness judgments ranges from .45 to .88( 14) i 
and for accuracy judgments from .27 to 183, (14) (12) If evaluations of 
their previous performance are fed back to photo interpreters, their sub- 
sequent confidence estimates of accuracy are significantly improved, (21) 

One problem with this, however, is that such feedback is not available in 
actual photo iiverpeSbeta bie cuatitme . 

A similar but more sophisticated technique is to eeera several’ 
performance measures, such as time to first target detection, as well as 
confidence ratings, and bimpuité probabilistic ratings of accuracy and 


completeness which are fed back to the interpreters. Use of this pro- 


cedure has been found to reduce the number of subsequent incorrect * 





identifications, but did not significantly improve accuracy, complete- 
ness, or speed of identification. (14) If reliable confidence ratings 

ean be obtained, a decision rule must be formulated to determine which 
imagery is to be check interpreted. No experimental work has yet been 


attempted to select such an optimal decision rule. 
IV. TRAINING OF INTERPRETERS 


R. N. Colwell's artiele(?) summarized some of the methods currently 
used in training photo interpreters. H, W. Letbowttnt 25) has advocated 
using the programmed instruction technique. He argued convincingly from 
results of experimental psychology in perceptual learning that program- 
med instruction would be significantly more efficient than presently 
used lecture presentation. rca(18) conducted a special four-day training 
program in photo interpretation using tachistoscopic techniques similar 
to those used in reading improvement courses to increase interpreters' 
speed of detection and classification of targets. Comparisons between 
experimental and control group proficiency measures showed statistically 
significant performance improvement due to the special training. The 
experimental group extracted information from the test photography in 
one haif the viewing time required by the control group, with slight 
gains in campleteness and accuracy. Moreover, the effect of the training 
was such as to counteract deteriorating effects of diminished scale and 


inereased number of targets per photograph. 
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V. INTERPRETATION PROCEDURES AND WORKING CONDITIONS 


Work load can be expected to vary, especially at teen line" 
interpretation facilities. For conventional (non-stereo) large scale 
imagery, 60 ft./90 min. is ae | an acceptable low input quantity 
for average interpreters to view. A rate of 120 ft./90 min. would con- 
stitute a high input. (1) 

Performance has been found to fluctuate during the veraing day, 
but not in a consistent manner. A Pete ac Oe experiment suggested that 
there was no decrement in performance even during a work day of extended 
length, twelve hours, containing no rest periods and Taig ols 
short periials for lunch and henner. <2) The see Seany found low corre- 
lation between expressions of fatigue and performance. 

Interpreters can vary their performance as & function of the 
relative weights given to accuracy and elas een, WE unless they 
are given guidance, they will base their work methods on their own sub- 
jective and highly variable conception of the intelligence objectives. (20) 

Another study 2) found that completeness increased generally with 
increased viewing time. The study suggested that, given large quantities 
of photography on which features are #0 be identified accurately, one 
minute viewing time past photograph yielded high performance. A 48 eid 
utes on and 5 minutes off work-rest cycle vas recommended. 

Aero Service Coxpdyatien™? found that when short (25 ft.) samples 
of imagery were interpreted, an acceptable methodology was to proceed 


directly to interpretation without first rapidly screening the imagery. 


Willmorth and Birnbaum (22) (23) recommended that neither screening nor * 


overlapping of imagery be used for rapid interpretation. 
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Investigating interpreter team organization, Bolin, Sadacca, 
and, Martinex‘©? found no single factor or principle of team organization 
that led to improved performance in all types of missions. Continuing 
this line of research, Doten, Cockrell, and Sadacca‘ +2) found that teams 
in which the check interpreter had complete knowledge of the initial 
interpreter's work produced more complete results with higher efficiency 
than did procedures utilizing only partial knowledge of initial interpreta- 
tion. Arbitrary checking (where the checker made final judgments without 
consulting the initial interpreter), consensus checking (where only 
those interpretations agreed upon without discussion were recorded), 
and discussion-consensus checking (where only those interpretations agreed 
upon after discussion were recorded) procedures were tested. Introduction 
of a third man provided more completeness but reduced efficiency. No 
differences in team output from different procedures with the three-man 
team were noted. The checking procedure with arbitrary scoring resulted 
in the highest completeness but lowest accuracy. Checking procedure 
with consensus yielded higher accuracy but less complete interpretation. 
Discussion with consensus scoring gave both high accuracy and completeness 


but reduced efficiency. 


eal 





VI. RELATION TO PREVIOUS RESEARCH 

No analyses based on operational data were found in the litera- 
ture on human factors research in image interpretation. Most studies 
conducted to date have used advanced photo interpreter trainees as : 
subjects in controlled experiments. The all-but-insurmountable diffi- 
culty encountered when attempting to use operational environments as 
sources of data is the measurement of interpreter performance. More 
precisely, the difficulty is in defining the imagery's ground truth. 
All standard measures of interpreter performance--accuracy, complete- 
ness, conciseness (the ratio of accuracy to time), and efficiency-- 
are dependent upon ground truth. Definition of ground truth is a 
tedious process, generally accomplished by consensus decision follow- 
ing careful interpretation by a team of expert photo interpreters. 
Nevertheless, if ground truth is known, it should be possible to insert 
that imagery between sequences of operational imagery. Such a procedure - 
might provide more reliable indication of operational interpreter per- 
formance than that obtained from presently employed procedures. 

This study was constrained by lack of trained image interpreters; 
however, it was felt that the important factors in team studies would 
be found in experiments using untrained subjects. 

Simulated photo imagery was constructed because (1) its campo- 
sition could be controlled exactly, (2) ground truth could be deter- 
mined easily, and (43) symbolic targets could be used. Untrained subjects 
would be expected to find identification of objects in aerial photographs 


inordinately difficult, on account of their lack of experience in iden- 


tifying objects from vertical or high oblique aspects. It was felt 








that use of more familiar symbolic targets would result in a better 
balance of identification, classification, and evaluation difficulties 
for untrained subjects than would use of actual aerial photography. 
Test image interpretability was designed to be dependent upon target 
density, shape, markings , scale, contrast with background, resolution, 
detail, and spatial location, as well as background noise. 

Experiments I-and II were meant to complement the APRO team 
studies cited. Experiment II, in addition, was designed to complement 


the pre-processing studies cited. 
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CHAPTER III 


EXPERIMENTAL DESIGN AND RESULTS OF THREE EXPERIMENTS 


ON IMAGE INTERPRETATION PERSONNEL ASSIGNMENT CRITERTA 
I. ELEMENTS COMMON TO ALL EXPERIMENTS 


A Preliminary Experiment was conducted in order to provide data 
necessary for Experiments I and II, each of which was directed toward 
one of the primary objectives of the present ‘study. Certain methodolog- 
ical elements were common to the three experiments. 

Interpretation Tasks 

The interpretation tasks used in the study consisted of two sub- 
sets of activities. These were: 

1. Initial interpretation. Interpreters worked independently 
On separate parts of the imagery, completing annotations and target 
identifications. 

2. Checking. Interpreters checked their teammates’ initial 
interpretations and looked for additional targets. 

Team Scoring Rules 

A scoring rule was defined as a means of combining individual 
output into a team output. The two basic scoring rules were: 

1. Arbitrary. Score all responses which checkers approve or 
make. 


2. Combined. Score and sum all responses which both teammates 


make using the initial interpretation procedure. 





Dependent Variables. 


Three measures of individual interpreter performance were used: 
1. Accuracy, Ratio of right interpretation to the sum of right 
plus wrong interpretations. 
a. Completeness. Ratio of right interpretations to the total 
possible rights, i.e., the total number of scored targets in the imagery. 
3. Efficiency. Number of right interpretations divided by the 


total amount of time required in minutes, 


: Experimental Subjects 


Twenty-four Army and Marine Corps officers enrolled in the 
Operations Research program at the Naval Postgraduate School constituted 
the population of subjects for the three experiments. Their sore dis- 
tribution was: 12 captains, 10 majors, and @ lieutenant colonels. One 
subject was a pilot. One subject had previous experience in photo inter- 


pretation. 


Subject Proficiency 


It was assumed that the subjects had acquired some degree of 


proficiency in detection and classification tasks other than image 





interpretation which would give them individually varying proficiency 
in image interpretation tasks. It was further assumed that the sub- 
jects" proficiencies could be measured and the subjects ordered accord- 
ing to those proficiency measurements, and that this ordering would 
not change during the course of experimentation, due to learning 


or any other cause. 
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Experimental Imagery : 
Sixty 7" x 8" image frames were hand drawn on 8 1/2" x 11" white 
paper, These image frames were assembled into six different ten frame 
imagery sets. Twenty-four Xerox copies of each set were made; this 
quantity was sufficient to insure that no subject would view any of the : 
‘imagery more than once during the course of the experimentation. Orig- 
inal copies of the imagery were drawn in red, black, blue, and green 
ink; this produced controlled differences in contrast ratios in the 
Xerox copies. | 
The image frames were intended to simulate photographs of targets 
in the vicinity of a border between two countires. Fifteen classes of 


targets were represented. Appendix A contains samples of the test imagery. 


II. PRELIMINARY EXPERIMENT: MEASUREMENT OF INTERPRETER 


PROFICIENCY AND IMAGERY DIFFICULTY 


Experimental Objectives 


The objectives of the Preliminary Experiment were (1) to measure 
interpreter proficiency, (2) to measure imagery difficulty, and (3) to 
determine if there was any significant difference in difficulty among 
the six sets of imagery used. 


Dependent Variables 


Individual interpreter proficiency was calculated from individ- 


ual accuracy, completeness, and efficiency scores according’ to the 





following formula: 


Proficiency, = 1/3 (Accuracy, + Completeness, + Normalized Efficiency, ) 


Where 


Normalized Efficiency, = Efficiency, - min (Efficiency, ) 
all 





max (Efficiency, ) - min (Efficiency. ) 
i 1 qd: a 


t= 1,2,...,24 


Image frame difficulty was determined according to: 


Difficulty, = 1 - 1/2 (Mean Accuracy Frame. + Mean Completeness Frame_) 
J J J 


Experimental Design 


The experimental design to test effects of different imagery sets 
on interpreter performance is shown in Figure 1. Assignment of imagery to 
subjects was random, subject to the balance requirements that (1) each 
subject interpret two different imagery sets, and (2) each imagery set 


be interpreted by eight subjects. 


Experimental Procedures 

Rach subject was given two imagery sets of ten image frames each. 
A separate interpretation key, showing examples of each type of target, 
was provided. The interpreter was required to circle or draw an arrow 


to each target detected and label each with a number. The numbers were 
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PERFORMANCE MEASURE 
Completeness Accuracy 


Efficiency 





Set D 


IMAGERY 


Figure 1. Preliminary Experiment Design 





then entered on appropriate lines on the target identification form 
printed below the image frame. 

After an interpreter completed his first set of ten image frames, 
he recorded the time, measured in 15 second increments, and commenced 
work on the remaining ten image frame set immediately. On completing 
the entire twenty image frames, total time was recorded. Interpreters 
were instructed to work independently, without going back to campleted 
frames, pacing themselves in anit to maximize their accuracy, complete- 


ness, and efficiency scores, 


Results 

Accuracy, completeness, and efficiency scores for each subject 
were tabulated and are presented in Table I. From these, proficiency 
scores were calculated,and subjects were ranked in order of decreasing 
proficiency scores, as shown in Table IT. 

Image frames were ranked in order of decreasing computed diffi- 


culty within each imagery set, as shown in Table III. 


¢ 


Analysis of variance. A 6 x 3 factorial analysis of variance was 
performed using the data summarized in Table I. Results of the analysis 
of variance are shown in Table IV. All tests of hypotheses were made 
at a five per cent significance level. Differences due to performance 


measures were statistically significant. Differences due to imagery 


sets were not significant. Interaction between imagery sets and 


performance measures was not significant. 





TABLE I 


ACCURACY, COMPLETENESS, AND EFFICIENCY SCORES FOR SUBJECTS 
IN THE PRELIMINARY EXPERIMENT s 





Imagery Ac¢uracy Completeness Efficiency 
j 
Set A 981 684 4.952 
1.000 .829 5.040 
.972 .897 4.667 
.986 S47 5.055 
.928 B42 5.22) 
.970 B42 7.758 
1.000 .921 7110 
1.000 .868 8.000 
Set B 984, »795 6.359 
.848 -719 3.672 
.932 sop . 5.520 
«986 923 5.878 ~ 
-969 195 5-905 
-969 .808 6.811 


.983 mM he 8.769 





TABLE I (continued) 


————————————————————————————————————————E—EE—————E—EEEEEE————EE 








Imagery Accuracy Completeness Efficiency 
Set E «950 a9e9 5.098 
938 2779 286 
Obe -779 4.898 
973 oe 7.100 
~970 ~ 8h 6.341 
1.000 » Sy 4.906 
1.000 ~909 6.829 
1,000 .870 5.360 
Set F 969 539 4. 82h 
U7 <9ah e897 
973 93h 5.680 
986 - 908 7.459 
- 933 137 4.148 
~962 «671 34778 
956 855 4.561 


984, 829 8,129 
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TABLE If 


RANKING OF SUBJECTS IN ORDER OF DECREASING FROPICTENGY, 
BASED ON PRELIMINARY EXPERIMENT DATA 


Subject Number Proficiency Rank 
1 .606 22 
2 .853 5 
3 - 706 I 
4 955 i. 
5 MOTD. 4 
6 .612 pa. 
7 - 790 10 
8 -T40 Ha 
9 .697 18 

10 ai bel 14 
11 .652 20 
12 .602 23 
Pine: -719 16 
14 47 oh 
LD -739 12 
16 .810 8-9 
if .820 q 
18 927 2 
19 1138 15 
20 .926 3 
21 674 19 
22 .810 8-9 
23 831 6 
ah ~720 15 





TABLE TIT 





RANKING OF IMAGE FRAMES WITHIN SETS IN ORDER OF DECREASING DIFFICULTY, 
BASED ON PRELIMINARY EXPERIMENT DATA 


4. Frame Difficulty Rank 


#030 i 
. 082 
039 
«055 
. 223 
.078 
. 069 
.O31 
. 063 
065 
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.076 
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.162 
Pe n=. 
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i i 050 8 
‘i 2 .063 6 
: ) 088 mM 
4 4 068 5 
: Set C 5 . 062 7 
6 .096 3 
; 7 .113 2 

8 .290 a 

9 .000 10 

10 .Oh2 9 
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TABLE IIL (continued) 
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TABLE IV 


PRELIMINARY EXPERIMENT 
ANALYSIS OF VARIANCE 








Source af ss MS F 





Performance Measure 2 764. 245 382.123 
Imagery 5 1.216 0,243 0.356 
PM x I 10 2.445 0.245 0. 359 
Residual 126 85.934 0.682 





Total, | 13 853.840 
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Discussion and conclusions. The formula used to calculate inter- 
preter proficiency was selected arbitrarily to yield proficiency scores 
in the unit interval [0,1]. This formula weighted efficiency more highly 
than accuracy or completeness, which did not seem unreasonable. The rel- 


ative importance of accuracy, completeness, and efficiency in the real 


world can be expected ‘to vary with changing tactical_and strategic image 


interpretation requirements; hence, no one formula for proficiency can 


nd 


be said to be best for all situations. Likewise, the formula for. calcu- 


lating image difficulty was chosen arbitrarily to yield scores in the 


unit interval ia ay 
Data on performance of interpreters using different sets Seat 
indicated that use of any particular set of imagery did not bias an inter- 
preter's performance scores relative to those of other interpreters. 
There was no significant interaction between imagery and performance 
measurements. The importance of these results was that it permitted com- 
parisons of scores among interpreters using dissimilar test imagery. The 
significant main effect due to performance measures indicated that the 


performance measure factor should be included in the design of Experi- 


ments I and II. 


TIT. EXPERIMENT I: PROFICIENCY AS THE CRITERION 


FOR ARBITRARY CHECK PROCEDURE PERSONNEL ASSIGNMENTS 


Experimental Objective 
The objective of Experiment I was to determine if either of two 
personnel assignment methods using interpreter proficiency as the assignment 


eriterion would yield significantly improved team performance. 
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Personnel Assignment Methods 

The following. three personnel assignment methods were employed: 

1. lLow-initial/High-check. Initial interpretation was performed 
by the lower proficiency team member. Checking was done by the higher 
proficiency team member. 

2. High-initial/Low-check. Initial interpretation was performed 
by the higher proficiency team member. Checking was done by the lower 
proficiency team member. 

4. Random initial/cheek. Initial interpretation and check inter-~ 
pretation personnel assignments were made without regard to interpreter 
proficiency. 

All personnel assignment methods were scored according to the 


arbitrary scoring rule. 


Experimental Design 


Experiment I design, to test effects of proficiency as a criterion 
for check task assignments, is shown in Figure 2, Subjects were assigned 
to two-man teams in restricted randomized fashion, subject to the re- 
quirements that (1) subjects who interpreted any of the same imagery 
in the Preliminary Experiment were ineligible for membership on the 
same team, and (2) each team was composed of one subject from among the 
twelve most proficient interpreters and one subject from among the 
twelve least proficient interpreters. Restriction (1) was necessary 
in order to use imagery annotated in the Preliminary Experiment as 


material to be checked in Experiment I without any subject's checking 


ll 





ASSIGNMENT CRITERION 


Low-Initial/High-Check High-Initial/Low Check 





Random Initial/Check 





Accuracy Completeness Efficiency 


PERFORMANCE MEASURE 


Figure 2. Experiment I Design. 
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imagery he had interpreted previously. Data generated during the 
Preliminary Experiment test session and during the Experiment I test 
session were combined, so that, in effect, each team interpreted the 

same imagery using both the Low-initial/High-check and High-initial/Low- 
check procedures. Restriction (2) was designed to accentuate any differ- 
ences between the two check procedures by encouraging wider range in 
proficiency between team members, Teams were grouped so that Group 1 
consisted of eight subjects using High-initial/Low-check es 

Group 2 consisted of eight subjects using Low-initial/High-check procedure, 
and. Group 4 consisted of eight subjects using the random initial/check 


procedure, This grouping is graphically illustrated in Figure 3. 


Experimental Procedures 


A group testing session was held one day after the Preliminary 
Experiment, during which session both Experiment I and Experiment II were 
conducted. 

Each subject was given the two imagery sets that had been inter- 
preted in the Preliminary Experiment by his teammate. Subjects were 
instructed to check their teammates’ interpretations, making corrections 
when appropriate, and to interpret additional targets missed by the 
initial interpreters. Time was recorded after the first ten frames were 
checked and after completion of the entire twenty frames. Checkers were 
instructed to work independently without going back to completed frames, 
pacing themselves in order to maximize team accuracy, completeness, and 


efficiency scores. Separate interpretation keys were provided. 
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Preparation of 12 copies of each of 6 dif- 
ferent imagery sets in 36 booklets, each 
containing 2 different imagery sets 











2 imagery booklets 


Preliminary Experiment Test Session 


2h interpreted booklets 





Scoring of interpreted imagery; calculation of 


subject proficiency and image frame difficulty 





Ss proficiency scores 


Assignment of subjects to teams, each team 
consisting of one subject from the set of 12 
2 inter- most proficient subjects and one subject from 
preted booklets | the set of 12 least proficient subjects 








2h Ss preted poi teams 


\ 
Assignment of subjects to groups for Experiment I - 





8 high proficiency 8 low proficiency low proficiency Ss3 \ 
Ss; 8 booklets in- Ss; 8 booklets in- booklets interpreted in 
terpreted in Prelim terpreted in Prelim Prelim Exp by high pro- 
Exp by low profi- Exp by high profi- ficienty teammates; ) 

ciency teammates ciency teammates high proficiency Ss; 


booklets interpreted in 
Prelim Exp by low pro- 
ficiency teammates 





Experiment I Test Session 


Figure 3. Experiment I Flow Chart. 





Results 
pinta d a AES 


Team accuracy, completeness, and efficiency scorés are shown in 
Table V. Table VI presents initial interpretation scores and incre- 


mental scores resulting from check interpretation, 


: sf 
Analysis of variance. A 3 x 3 factorial analysis of variance was 


performed. Data used are presented in Table V. Results are shown in 
Table VII. All tests of hypotheses were made at a five per. cent signifi- 
cance level. Differences due to performance measures were statistically 
significant. Differences due to personnel assignment criteria were not 
significant. Interaction hetween personnel assignment criteria and 
performance measures ee significant. 

Diseyssion and conclusions. Data on performance of interpretation 
teams using different personnel assignment criteria indicated that none 
of the three criteria was.to be preferred to any other. This was an un- 
expected result, for it, had been assumed that the Low-initial/High-check 
procedure would prove superior to the .other procedures. An attempt was 
made to account for this result. It was noted from the data in Table VI 
that mean low proficiency initial interpretation scores were below mean 
high proficiency initial interpretation scores; the Low/ High ratios for 
accuracy, completeness, and efficiency were .985, .948, and .732, respect- 
ively. In addition, the means of low proficiency incremental scores-due 
to checking were below those of high proficiency incremental checking 
scores; the Low/High ratios in this case were .188, .505, and .667, respect- 
ively. Team performance was determined by combining initial and incre- 


mental checking. scores. 
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TABLE V 


ACCURACY, COMPLETENESS, AND EFFICIENCY SCORES FOR TEAMS 
IN EXPERIMENT I rs 

















Procedure Accuracy Completeness Efficiency 
973 he 3.058 
.979 .922 he Qk1 
mo)ls,) .868 4.000 
Hi gh-Initial 971 one 3.750 
Low-Check 943 -955 4 S9T> 
965 - 710 3-359 
1.000 Gb 4 £028 
992 857 4.981 
-960 . 783 3.748 
986 -91T 3.623 
979 -960 3.425 
Low+ Initial 986 -910 53.639 | 
High+eCheck 925 ETT 2.941 
-961 a G50 .- tao 
.973 gh2 3.314 
957 859 3.646 
-960 G2 4.920 
. 987 -925 3,322 
.980 he 3, 2h0 
Random Initial .980 947 3.972 
Check -960. 928 3.337 
. 966 .928 4.028 
986 915 3.256 


-973 .922 4.000 











Accuracy Completeness Efficiency Accuracy Completeness Efficiency 
-Inecrement Increment Increment 














.892 748 3.966 .021 . 026 - 2.068 

.938 ATG 5.545 - .021 . O46 - 2:631 

1930 5 4,372 * , 06D 072 = 2,382 

.978 .865 5.414 . 000 -013 - 2.666 

969 612 4.895 - .005 - .116 - 3.205 

Lower 961 925 5.026 = Ol 029 = 2,902 

Half 939 909 5,105 000 026 - 4.722 

932 795 4.863 008 065 - 2.685 

972 896 5, 3508 015 O46 - 2.039 

971 906 4 538 - .006 052 - 2.140 

979 892 4 828 OT 026 —.1.520 

E 983 758 4. 336 - .O14 020 - 3.019 
Mean : 

Low 954 822 4.855 003 O45 - 2.665 

+959 916 5.308 033 129 =» 1,085 

1.000 876 6.872 O48 1ho = 1.922 

1.000 869 6.410 030 145 - .990 

992 Bib 7.647 008 O45 m 1.775 

970 826 6.564 - .009 val - 1.147 

Higher 955 826 Woot 026 000 - 1.704 

Halt 973 916 7.780 O54 033 — LeO71 

931 803 6.685 025 064. - 1.217 

951 882 6.067 - .012 O46 - 488 

986 895 6.112 008 054. - 1.083 

926 896 5.520 - .018 038 - 1.289 

985 857 6.769 003 157 - 1.080 

" Mean — ; 
High -969 867. 6.634 2015 085 - 1.299 





TABLE VI 
INITIAL INTERPRETATION SCORES ; 
AND INCREMENTAL SCORES DUE TO CHECKING 








Source 


Performance Measure 
Assignment Criterion 
PM x AC 

Residual 


Total 


TABLE Vil 





EXPERIMEN? I 
ANALYSIS OF VARIANCE 


af SS 
2 125.823 
2 0.598 
4 0.856 
63 15.105 
71 142. 380 


MS 


62.912 
0.299 
0.214 
0.2h0 
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fhe entries in Table VIII were obtained by adding mean low proficiency 

initial scores to mean high proficiency incremental check scores and by “y 
adding mean high proficiency initial scores to mean low proficiency check : 
incremental seores for accuracy, completeness, and efficiency. The ratios 

of the Low/High sums to the High/Low sums were .997, .995, and .896. 

These were closer to unity than were the initial ratios or the incremental 
ratios. Thus, checking served to balance out differences between high and 


low initial interpretations. Lest there by any temptation to conclude. 


_ from these figures that a Hi gh-initial/High~check procedure would yield 


significantly higher team performance, it should be noted here that Doten, 


Cockrell, and Sadacca(12) found the performance of High/Low proficiency 


‘teams (each man checking the other's initial interpretations ) to be better * 


than Hi gh/High proficiency teams. These two findings are not necessarily 
inconsistent because Experiment I yielded data on high proficiency check 
incremental scores to low proficiency initial scores from which nothing 
ean be deduced about high proficiency increments to high proficiency 
initial scores. If the results of both the Doten study and Experiment I 


were valid, then {t would follow that mean Hi gh/ High incremental scores 


could be expected to be less than mean Low/Hi gh incremental scores. 
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TABLE VIII 


INITIAL AND INCREMENTAL PERFORMANCE RATIOS 














Mean Scores Accuracy Completeness Efficiency 
Mean Low Initial 2954 822 4.855 
Mean High Initial .969 . 867 6.634 
Mean Low tnitial 985 aoc 732 
Mean High 
Mean Low Increment -003 e055 - 2.665 
Mean High Increment +015 .085 - 1.299 
Mean LOW fncrement .188 £505 067 ' 
Mean High 


Mean Low Initial + 
Mean High Increment .969 .907 3.969 . 


Mean Low/Mean High .997 -995 .896 
Mean High/Mean Low 
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TV. EXPERIMENT II: INTERPRETER PROFICIENCY AND IMAGERY DIFFICULTY 


AS CRITERIA FOR ASSIGNMENT OF IMAGERY TO INTERPRETERS 


Experimental Objective 


The objective of Experiment II was to determine if assigning the 
more difficult imagery to the more proficient interpreter and the 
easier imagery bo the less proficient interpreter would result in signi- 
ficantly higher team performance than random assignment of imagery to 


team members. 


Imagery Assignment Methods 


The two imagery assignment methods were: 

1. Presorted. Each interpreter received equal quantities of imagery 
for interpretation. All imagery given to the lower proficiency team 
member was less difficult than any of the imagery given to the higher 
proficiency team member. 

2. Unsorted. Imagery was assigned to team members without regard 


to its difficulty or interpreter proficiency. 


Experimental Design 


Experiment II design, to test effects of interpreter proficiency 
and imagery difficulty as criteria for interpretation task assignments, is 
shown in Figure 4. Composition of the twelve teams ee the same as 
in Experiment I. Each team interpreted wnannotated imagery of known 
_ difficulty not previously viewed by either team member. Teams were 


randomly grouped such that Group 1 consisted of six teams using the 
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ASSIGNMENT CRITERION 


Unsorted 


Sorted 


Accuracy 





Completeness Efficiency 


PERFORMANCE MEASURE 


Figure l. 


Experiment IT Design. 
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Unsorted imagery interpretation procedure while Group 2 consisted of six 
teams using the Presorted imagery interpret«tion procedure. Team grouping 


and imagery flow is shown in Figure 5, 


Experimental Procedures 


Interpretation procedures were similar to those of the Preliminary 
Experiment, except as noted. Each team was given twenty image frames to 
interpret. Those using the Presorted assignment method were given imagery 
in ten frame presorted booklets. Those using the Unsorted assignment 


method were given a stack of twenty unsorted imagery frames with team 


"members being instructed to take an image frame off the top of the stack 


after each frame was interpreted, until the stack was exhausted. Each 


interpreter recorded the time when he finished all the imagery assigned 


to’ him. Interpreters were instructed to work independently without going 
back to completed frames, pacing themselves in order to maximize team accu- 
racy, completeness, and efficiency scores. Users of presorted imagery were 


not told the imagery was presorted. 


Results 
Accuracy, completeness, and efficiency scores for each team were 


tabulated and are presented in Table IX. 


Analysis of variance. A 2 x 3 factorial analysis of variance was 


performed. Results of the analysis are shown in Table X. All tests of 


_ hypotheses were made at a five per cent significance level. As before, 


differences due to performance measures were statistically significant. 


hg 








Preparation of 12 copies of each of 6 dif- 
ferent imagery sets in 36 booklets, each 
containing 2 different imagery sets 












2h°Ss 


12 imagery booklets 







Ss proficiency scores 


Assignment of subjects to teams, each team 
consisting of one subject from the set of 12 
imagery most proficient subjects and one subject from 
difficulty |the set of 12 least proficient subjects 
scores 





24 Ss assigned to 12 teams 


eo \T/ 
Assignment of subjects to groups for Experiment IT 





6 teams; 6 booklets 6 teams; 6 booklets 
(each low proficiency member 
to interpret less difficult 
half of booklet; each high 

proficiency member to inter- 
pret more difficult half of 
booklet) 


\ N 
Experiment II Test Session 


Figure 5. Experiment II Flow Chart. 











TABLE IX - 


ACCURACY, COMPLETENESS, AND EFFICIENCY SCORES FOR TEAMS 


Procedure 


Unsorted 


Pre-sorted 


IN EXPERIMENT IT 


Accuracy 


.960 
.980 
.960 
.965 
- 993 
moyel 


-993 
973 
£965 
972 
. 986 
980 


Completeness Efficiency 
. 774 7.619 
,955 7.688 
ORL 7.526 
902 B,. 118 
962 7.023 
.882 6.022 
954 te2¢2 
.922 7.889 
-890 7.667 
.890 7.211 
.903 6.829 
961 7.840 

oul 


Source 


Performance Measure 
Assignment Criterion 
PM x AC 


Residual 


TABLE X- 


EXPERIMENT II 
ANALYSIS OF VARIANCE 


af ss MS 
2 333.880 
1 0.028 0.028 
2 0.031 0.016 
30 3.542 0.118 
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Differences due to task assignment criteria for accuracy, completeness, 
and efficiency were not significant. Intersction between task assign- 


ment criteria and measures of team performance was not significant. 


Discussion and conclusions. Data on team performance using dif- 
ferent task assignment criteria indicated that neither the Presorted 
method nor the Unsorted method was to be preferred. If Presorting in- 
volved additional cost, the Unsorted method would be preferred. Results 
of this experiment suggested that development and subsequent procurement 
of equipment Sere ivcesea tmepery by predicting image difficulty would 


not be cost effective. 
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CHAPTER IV 
OPTIMAL ASSIGNMENT OF PERSONNEL TO IMAGE INTERPRETATION TASKS 2: 


A linear integer programming formulation of the problem of optimal 
utilization of personnel was developed and is presented in this Chapter. 
The solution is dependent upon knowledge of several assumed constant 
terms; these are: 

_1. Number of each class (high proficiency and low proficiency) of 
interpreters available; | 

2. Flow of imagery into the system, expressed in expected number 
of targets per unit time; 

3. %Expected efficiency of each class of interpreter for initial 
interpretation, for check interpretation of initial work done by an 
interpreter of his class, and for check interpretation for initial work 
done by an interpreter of the other class; and 

hk, Expected performance measures (accuracy or completeness) of 
each class of interpreter corresponding to the various efficiency measures. 

The system can be depicted in the flow ahart format of Figure 6, 
where 

1. X, is the number of class i interpreters available, i = 1,2; 

2. xX; is the number of class i interpreters utilized as initial 
interpreters, i = 1,2; 


De Xj is the number of class j interpreters utilized as checkers of 


class i initial interpretations, i =1,2, j = 1,2; 








Figure 6. Image Interpretation Flow Chart. 
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4, p is the flow of interpretable targets into the system; 

5. $1 is the flow of interpretable targets in arc 1, eS Lge 

6. ey is the efficiency of class i interpreters utilized as 
initial interpreters, i = 1,2; 

Tas ej is the efficiency of class j interpreters utilized as 
checkers of class i initial interpretations, i =1,2, j = iis 

8. ey is the expected completeness (accuracy) score of a class i 
interpreter utilized as an initial interpreter, i = 1,25 

9. C13 is the expected completeness (accuracy ) score of a class j 
interpreter utilized as a checker of calss i initial interpretation, 
tS ae 29 j= 3425 ane 

10. Uis the flow of uninterpreted interpretable targets in excess» 
of system capacity. 

This was written in linear program format as: 
MAX 3 (cz + c11) + Phe, + B5 (c1 + c12) + 6 (c2 + car) : 

+ Pree + $8 (ca + G22) 
Subject to 

p=U+ py + Po 


py & ex) 


Po & epXp 
, = bz + Py + Bs 
bo = Og + by + hg 
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pg = 


11 *17 
€12%10 
€01%07 


epaxoo 


x7] + X11 + X01 = Xy 


Xo + X00 + X12 = Xo 


X71 5X05 (x;o+ Xoo), (Xo1+ X11) non-negative integer 


2; Po,bi0,Po1, Pir, foo = 0 


The (xjo + Xoo), (Xo, + X,1) non-negative integer constraint, rather than 


X112X1 09X91 »Xo0 non-negative, was necessary in order to permit the pos- 


sibility of one checker serving both high and low proficiency initial 


interpreters. 


The problem can be written in terms of the x's only. Adding slack 


variables to the inequality constraints, 


fy +S) = e7 xX] 


fo + Sp = epXo 


Os +85 = e11%11 


fs + 85 = €1 9X19 


fe + SE = en)X ay 


fg + Sg = epoXop 


at 


solving for the f's 


#1 = ex, - Sz 


fo = eaxe - Se 

63 = e11x11 - 83 

$5 = e1ox10 - 85 

£6 = e21x21 - 6 

$8 = e20x20 - SB 

noting that 

py = fr - ps - ps5 
b7 = fo - £6 - £8 


The objective function can be written 
MAX (e71x11 - $3) (c2 + c]1) + e1x1.- Si - (e11x11- 83) - (elame - 85))cz 
+ (eyox 12- $5) (c1 + ¢12) + (e21xB1 - 86) (ce +, c27) 
+ (eaxe - Sa - (e21x21 - 86) - (e2ax29 ~- 88)) e2 + (eaax20 - SB) (e2 4 20) 
which can be reduced to 
MAX ¢11¢11%11 + C1e1X] + c12€12x12 + ¢21e2]%21 + c2eax2 + cazeQ0K2 --¢11Sz 
- ¢18) - cyp85 - ea18¢ - CoS - epaS 
Eliminating the variable 6's, the constraints become: 

€1)X] + €oX0 - Sy - S + U= ) 

x7 te X11 + X07 = x] 

Xo + X09 + X10 = Xo 


X1»X0, (x,o + X90), (Xo) + X,1) non-negative integer 


U > 


0. 





CHAPTER V 
SUMMARY AND CONCLUSIONS 
I. SUMMARY 


Methods of improving image interpretation system output through 
use of interpreter proficiency as a criterion for making interpreter 
personnel assignments were investigated. A Preliminary Experiment was 
eonducted to determine subject proficiency and. imagery difficulty. 
Analysis of variance indicated that the imagery used was sufficiently 
homogenous that measures of interpreter performance based on interpre- 
tation of dissimilar imagery sets could be compared. Experiment I was 
designed to determine if either of two personnel assignment methods using 
interpreter proficiency as the assignment criterion would yield signi- 
ficantly improved team performance. Analysis of variance revealed no 
significant differences in performance due to either of the methods 
tested. Experiment IT was designed to determine if assigning the more 
difficult imagery to the more proficient interpreter would result in a 
significantly higher team performance than random assignment of imagery 
to team members. Analysis of variance indicated no significant differ- 
ences in interpreter performance due to either of the methods tested. 


The image interpreter personnel assignment problem was formulated 


as @ linear integer program. 








II. CONCLUSIONS 





Insofar as image interpretation operations resemble the experi- 

mental conditions of this study, the relative proficiency of image inter- 

preters need not be considered in making personnel task assignments. 

Subject to the same qualification, pre-sorting of imagery by predicted 

difficulty of interpretation with subsequent assignment of the more 

difficult imagery to the more proficient interpreters cannot be expected 

to result in improved system output. | 
If measures of expected interpreter proficiency and expected 

input rate of interpretable targets are known, optimal assignment of 


interpreter personnel can be made using an integer linear programming 


formulation of the problem. 
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APPENDIX C 
INSTRUCTIONS TO SUBJECTS 


The following instructions were given to subjects prior to the 
Preliminary Experiment: 

"The imagery you will be given is designed to represent aerial 
photography. The territory depicted shows a border between tuo countries 
called WHITE and BLACK. You are WHITE photo interpreters who have been 
given the task of detecting all BLACK military objects on WHITE's side 
of the border. 

"In the imagery the border is indicated by a line of x's. The 
WHITE side and.the BLACK side are clearly labelled. The border will 
often be unrealistically irregular in configuration. On the other hand, 
the military objects will often be unrealistically simple. WHITE and 
BIACK military objects are similar in appearance; they differ in that 
BLACK forces are drawn with portions shaded, whereas WHITE forces are 
drawn in outline with no shaded portions. You are not to report any 
WHITE forces--no matter which side of the border they are “ You 
are not to report any BLACK forces on BLACK's side of the porder. Re- 
port only those BLACK forces that have violated WHITE's territory. Is 
that iet (Wait for response. ) 

"You have an ievereerenidlich key betes! ven: It shows examples of 
the symbols you will see on the imagery. You may refer to the key as 
you interpret the imagery. The key lists the fifteen different types 


of military objects you are looking for. For your purposes no other 


types of military objects exist. The key is arranged in alphabetical 





order. Note that the types of military objects are: 

Aircraft, jet fighter 

Aircraft, prop fighter 

Aircraft, multiengine 

Airfield 

Building 

Helicopter 

Missile 

Radar Antenna 

Radio Antenna 

Road 

Tank 

Tent 

Trench - 

Truck, Long 

Truck, short 

"Please open your imagery booklet to the first page, labelled 
EXAMPLE 1. This image frame is similar to those in the rest of the 
booklet. In this frame BLACK's territory is roughly the upper right 
quadrant. There are seven military objects: 

one BLACK radio antenna 

two WHITE tents 

two BLACK tanks 


one BLACK trench 








one BLACK tent 
Recall, however, that you are a WHITE interpreter concerned only with 
reporting BLACK border violations. Therefore only the BLACK tent, tank, 
and trench are of interest to you, as they are located in WHITE territory. 

"Your task will be to look at each image frame, circle or draw 
an arrow to each BLACK border violator, assign a number to each violator, 
and write the numbers on the appropriate lines of the evaluation form 
below the image frame. Please turn to the next page for an example. 

"On the imagery the tent has been circled and numbered 1; the num- — 
ber 1 has been caer on the appropriate line, The tank is cireled and 
labelled 2; note the 2 on the line opposite 'tank.' An arrow is drawn 
to the trench, which is numbered 5; the number 5 appears opposite ‘trench' 
‘below. ‘Eee Grom any of the other objects would be scored as errors. 
Your choice of labelling numbers is immaterial, just as long as each num- 
ber used is not repeated on the same image frame. 

“Please turn the page to EXAMPLE 2. Mark all BLACK forces on 
WHITE’s side, and fill in the evaluation form below. Look up when you 
have finished. (Pause. ) 

' "Now turn to the next page. The long truck and two helicopters 
have been labelled and recorded below. Note the double entry on the 
helicopter line. “The BLACK tent is not recorded because it is on BIACK's 
side of the border. The black shape near the border in the upper part 


of the frame does not represent any military object. Note that the 


helicopter to the right contrasts less with the background than does 





the helicopter to the left. You can expect some objects to be difficult 
to see on account of low contrast ratio. You may have to guess the 
identity of an indistinet or lightly drawn object or guess if it g 


is shaded or not. It may be costly to WHITE for you to miss an in- 





truder; on the other hand, false alarms may also be costly. Do as well 

as you can. | 
"When I say START, turn to the next page and begin your image 

interpretation. Work as accurately, completely, and quickly as you 

ean. As soon as you finish one image frame, go immediately to the 

next. After ten frames you will come to an instruction page. When you 

reach it, record the time as indicated on the flip cards I have on 

the desk here in front, jand go on to the next set of frames immediately. 

After completing ten more frames, stop, and record the time from the 

flip cards. Do not look back at any image frame you have completed. 

You are then free to leave. You will be scored on accuracy, completeness, a 

and speed--so pace yourselves to maximize your score. Are there any 


questions? (Pause.) You may start in ten seconds. (Pause.) START." 


The following instructions were given to subjects prior to Experiment 


"You have each been| given an imagery booklet that was used by an 

initial interpreter in yesterday's experiment. It contains marked image 

frames and evaluation forms. Your task in this experiment is to check- 

interpret the imagery. You should correct any omissive or commissive 

errors you find. If you) find a commissive error, X through the initial - 


interpreter's marks and mark the frame according to your interpretation. 


Please do not erase any of the initial interpreter's marks--X through 








them. 





If you find omissive errors--that is, BLACK forces on WHITE's side of the 
border that were overlooked by the initial interpreter--label them and 
make appropriate entries on the evaluation form beneath the imagery. 
(Demonstrate on blackboard.} Is this clear? (Pause. ) 

"In this experiment you and the initial interpreter are considered 
a two man team. Your team will be scored on the basis of accuracy, com- 
pleteness, and speed. Your teammate has, in effect, already done the 
initial interpretation. You should not change his correct interpretations. 
Any corrections you make will be final judgments--that is, your teammate 
will not be checking your —— 

"When you are told to START, check-interpret the first ten image 
frames without stopping. Do not go back to a frame you have checked. 
When you reach the instruction page, record the time you see on the flip 
card here in front, and go immediately to the next set of image frames. 
When you finish, stop and record the time. Please remain seated until 
everyone finishes, | 

"We'll take a short break when everyone is finished. The final 
experiment--which is a short one--will follow the breek. Are there any 
questions? (Pause.) You may START in ten seconds. (Pause.) START." 

The following instructions were given to subjects prior to 
Experiment FT: 

"You have been assigned to teams and should be seated next to your 
teammate. Some of you have been given sets of ten image frames to 


interpret. Others of you have a stack of twenty image frames which should 


be placed within reach of both members of the team. 





"When you are told to START, you should interpret your image 
frames just as you did in yesterday's experiment. If you have your own 
individual booklet, work through the frames without stopping. When fin- 
ished, record the time from the flip cards. Those of you with a shared 
stack of imagery: When told to START, each team member should take one 
image frame from the top of the stack and interpret it. As soon as 
you've finished a fram=, take another from the top of the stack. Con- 
tinue working until your team has exhausted the imagery. Each team 
member should call out his number and record the time when he finishes 
his last frame. Please remain seated until everyone is finished. Are 
there any questions? (Pause.) You may START in ten seconds. (Pause. ) 


START. " 
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